Build CF host debian packages in parallel#2621
Conversation
GoogleCuttlefishTesterBot
removed
the
kokoro:force-run
| sudo docker build --file "tools/buildutils/cw/Containerfile" --tag "android-cuttlefish-build" . | ||
| sudo docker run -v=$PWD:/mnt/build -w /mnt/build -v=$HOME/bazel-disk-cache:/root/bazel-disk-cache android-cuttlefish-build base -d /root/bazel-disk-cache & |
There was a problem hiding this comment.
The standard GitHub runners claim to have only 4 CPUs allocated. On top of that, bazel and go build which are used for cuttlefish-base and cuttlefish-frontend already have parallelism built in. I believe npm install is the only build system that will run serially, while the others will expand to fill all available CPUs, without coordinating to share the CPUs with other build systems.
By default docker run will use CPU share constraints for giving CPU time out to instances, but will still report the number of available CPUs that the host has unless controlled with a cpuset.
There was a problem hiding this comment.
After looking at your comment, I'm leaning to close the PR. Though I still believe this PR could reduce build time as the rest can fulfill resource availability, now I have a bit of concern that cons(e.g. threat of OOM, worse for debugging, more chances to be flaky) might be bigger than pros.
3 participants
In
build-debian-package-{arch}GitHub workflow job, it tries to build debian packages for each directory in a row. This PR suggests to build CF host debian packages in parallel, and wait until all build procedure is done. I expect we can reduce presubmit/postsubmit workflow time with applying this PR.However, you can see the elapsed time of
build-debian-package-{arch}is huge. This is intended behavior, as the bazel cache is invalidated by changing corresponding GitHub action code. Once it's merged to the main branch and writing bazel cache via postsubmit procedure, I expect it would be reduced as intended.